Digital histopathology and microdissection

ABSTRACT

A computer implemented method of generating at least one shape of a region of interest in a digital image is provided. The method includes obtaining, by an image processing engine, access to a digital tissue image of a biological sample; tiling, by the image processing engine, the digital tissue image into a collection of image patches; identifying, by the image processing engine, a set of target tissue patches from the collection of image patches as a function of pixel content within the collection of image patches; assigning, by the image processing engine, each target tissue patch of the set of target tissue patches an initial class probability score indicating a probability that the target tissue patch falls within a class of interest, the initial class probability score generated by a trained classifier executed on each target tissue patch; generating, by the image processing engine, a first set of tissue region seed patches by identifying target tissue patches having initial class probability scores that satisfy a first seed region criteria, the first set of tissue region seed patches comprising a subset of the set of target tissue patches; generating, by the image processing engine, a second set of tissue region seed patches by identifying target tissue patches having initial class probability scores that satisfy a second seed region criteria, the second set of tissue region seed patches comprising a subset of the set of target tissue patches; calculating, by the image processing engine, a region of interest score for each patch in the second set of tissue region seed patches as a function of initial class probability scores of neighboring patches of the second set of tissue region seed patches and a distance to patches within the first set of issue region seed patches; and generating, by the image processing engine, one or more region of interest shapes by grouping neighboring patches based on their region of interest scores.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalApplication No. 62/411,290, filed Oct. 21, 2016, and U.S. ProvisionalApplication No. 62/557,737, filed Sep. 12, 2017. These and all otherextrinsic materials referenced herein are incorporated by reference intheir entirety.

INTRODUCTION

The present technology relates generally to histopathology, themicroscopic examination of tissue for the purpose of determining whetherthe tissue is diseased and/or studying diseased tissue. The tissue maybe removed from any part of the body including, for example, breastlumps, specimens of bowel, kidney, liver, uterus lining, lung, chest,lymph node, muscle, nerve, skin, testicle, thyroid, or the like.

This disclosed technology relates to identifying regions of interestwithin a digital image, for example, identifying foreground objects frombackground scenes, or identifying cancer cells within a digitalhistopathology image.

The tissue may be collected from a subject in multiple settingsincluding biopsy, surgery, or autopsy. After tissues are removed fromthe subject, they are prepared for chemical fixation by being placed ina fixative such as formalin to prevent decay of the tissue. The tissuesare then either frozen or set in molten wax. Sections of the tissues arethen cut and placed on slides

Once the tissue sections are on slides, a pathologist views the slidesthrough a microscope to determine whether the tissue is diseased and, ifdiseased, determine the stage of the disease. For example, a pathologistmay determine whether a breast lump includes breast cancer cells and, ifit does, a pathologist may determine the grade and/or stage of cancer.However, there is a technical problem with these determinations in thatthey are often unreliable, expensive, time consuming, and generallyrequire verification by multiple pathologists to minimize the likelihoodof false determinations, including false positives as well as falsenegatives.

Embodiments of the present invention solve the above technical problemand provide a technical solution of using neural networks and, morespecifically, convolutional neural networks, to determine whether tissueis likely to be diseased.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates a block diagram of a distributed computer system thatcan implement one or more aspects of an embodiment of the presentinvention;

FIG. 2 illustrates a block diagram of an electronic device that canimplement one or more aspects of an embodiment of the invention;

FIG. 3 illustrates an architecture diagram of an electronic device thatcan implement one or more aspects of an embodiment of the invention;

FIG. 4 illustrates a process carried out by an electronic device thatcan implement one or more aspects of an embodiment of the invention;

FIG. 5 illustrates layers of a convolutional neural network with a layermodified for use with an embodiment of the invention;

FIG. 6 illustrates a process carried out by an electronic device thatimplements one or more aspects of an embodiment of the invention;

FIG. 7 illustrates a 256×256 pixel patch of tissue to be processed by anelectronic device that implements one or more aspects of an embodimentof the invention;

FIG. 8 illustrates a 400×400 pixel patch of tissue to be processed by anelectronic device that implements one or more aspects of an embodimentof the invention

FIGS. 9A-9F illustrate diagrams showing a plurality of patches of tissueto be processed by an electronic device that implements one or moreaspects of an embodiment of the invention;

FIG. 10 illustrates a Conditional Random Field Model, which, in analternative embodiment, can be used in place of or in addition to atleast some steps of the embodiment of FIG. 6;

FIG. 11 illustrates a diagram showing a region of interest boundarygenerated by an electronic device that implements one or more aspects ofan embodiment of the invention;

While the invention is described with reference to the above drawings,the drawings are intended to be illustrative, and the inventioncontemplates other embodiments within the spirit of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

The present invention will now be described more fully hereinafter withreference to the accompanying drawings which show, by way ofillustration, specific embodiments by which the invention may bepracticed. This invention may, however, be embodied in many differentforms and should not be construed as limited to the embodiments setforth herein; rather, these embodiments are provided so that thisdisclosure will be thorough and complete, and will fully convey thescope of the invention to those skilled in the art. Among other things,the present invention may be embodied as devices or methods.Accordingly, the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment, or an embodimentcombining software and hardware aspects. The following detaileddescription is, therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrases “in one embodiment,” “in an embodiment,”and the like, as used herein, does not necessarily refer to the sameembodiment, though it may. Furthermore, the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment, although it may. Thus, as described below, variousembodiments of the invention may be readily combined, without departingfrom the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” includes pluralreferences. The meaning of “in” includes “in” and “on.”

It is noted that description herein is not intended as an extensiveoverview, and as such, concepts may be simplified in the interests ofclarity and brevity.

All documents mentioned in this application are hereby incorporated byreference in their entirety. Any process described in this applicationmay be performed in any order and may omit any of the steps in theprocess. Processes may also be combined with other processes or steps ofother processes.

FIG. 1 illustrates components of one embodiment of an environment inwhich the invention may be practiced. Not all of the components may berequired to practice the invention, and variations in the arrangementand type of the components may be made without departing from the spiritor scope of the invention. As shown, the system 100 includes one or moreLocal Area Networks (“LANs”) / Wide Area Networks (“WANs”) 112, one ormore wireless networks 110, one or more wired or wireless client devices106, mobile or other wireless client devices 102-106, servers 107-109,optical microscope system 111, and may include or communicate with oneor more data stores or databases. Various of the client devices 102-106may include, for example, desktop computers, laptop computers, set topboxes, tablets, cell phones, smart phones, and the like. The servers107-109 can include, for example, one or more application servers,content servers, search servers, and the like.

Optical microscope system 111 may include a microscope, an ocularassembly, a camera, a slide platform, as well as components ofelectronic device 200 as shown in FIG. 2. Although FIG. 2 shows opticalmicroscope system 111 being communicatively coupled to server 109, itmay also be coupled to any or all of servers 107-109, network 112,wireless network 110, and/or any of client devices 102-106.

FIG. 2 illustrates a block diagram of an electronic device 200 that canimplement one or more aspects of systems and methods for interactivevideo generation and rendering according to one embodiment of theinvention. Instances of the electronic device 200 may include servers,e.g., servers 107-109, optical microscope system 111, and clientdevices, e.g., client devices 102-106. In general, the electronic device200 can include a processor/CPU 202, memory 230, a power supply 206, andinput/output (I/O) components/devices 240, e.g., microphones, speakers,displays, touchscreens, keyboards, mice, keypads, microscopes, GPScomponents, etc., which may be operable, for example, to providegraphical user interfaces.

A user may provide input via a touchscreen of an electronic device 200.A touchscreen may determine whether a user is providing input by, forexample, determining whether the user is touching the touchscreen with apart of the user's body such as his or her fingers. The electronicdevice 200 can also include a communications bus 204 that connects theaforementioned elements of the electronic device 200. Network interfaces214 can include a receiver and a transmitter (or transceiver), and oneor more antennas for wireless communications.

The processor 202 can include one or more of any type of processingdevice, e.g., a Central Processing Unit (CPU), and a Graphics ProcessingUnit (GPU). Also, for example, the processor can be central processinglogic, or other logic, may include hardware, firmware, software, orcombinations thereof, to perform one or more functions or actions, or tocause one or more functions or actions from one or more othercomponents. Also, based on a desired application or need, centralprocessing logic, or other logic, may include, for example, a softwarecontrolled microprocessor, discrete logic, e.g., an Application SpecificIntegrated Circuit (ASIC), a programmable/programmed logic device,memory device containing instructions, etc., or combinatorial logicembodied in hardware. Furthermore, logic may also be fully embodied assoftware.

The memory 230, which can include Random Access Memory (RAM) 212 andRead Only Memory (ROM) 232, can be enabled by one or more of any type ofmemory device, e.g., a primary (directly accessible by the CPU) orsecondary (indirectly accessible by the CPU) storage device (e.g., flashmemory, magnetic disk, optical disk, and the like). The RAM can includean operating system 221, data storage 224, which may include one or moredatabases, and programs and/or applications 222, which can include, forexample, software aspects of the digital histopathology andmicrodissection system 223. The ROM 232 can also include BasicInput/Output System (BIOS) 220 of the electronic device.

Software aspects of the digital histopathology and microdissectionsystem 223 is intended to broadly include or represent all programming,applications, algorithms, software and other tools necessary toimplement or facilitate methods and systems according to embodiments ofthe invention. The elements of systems and methods for interactive videogeneration and rendering program may exist on a single server computeror be distributed among multiple computers, servers, devices orentities, which can include advertisers, publishers, data providers,etc. If the systems and methods for interactive video generation andrendering program is distributed among multiple computers, servers,devices or entities, such multiple computers would communicate, forexample, as shown on FIG. 1.

The power supply 206 contains one or more power components, andfacilitates supply and management of power to the electronic device 200.

The input/output components, including Input/Output (I/O) interfaces240, can include, for example, any interfaces for facilitatingcommunication between any components of the electronic device 200,components of external devices (e.g., components of other devices of thenetwork or system 100), and end users. For example, such components caninclude a network card that may be an integration of a receiver, atransmitter, a transceiver, and one or more input/output interfaces. Anetwork card, for example, can facilitate wired or wirelesscommunication with other devices of a network. In cases of wirelesscommunication, an antenna can facilitate such communication. Also, someof the input/output interfaces 240 and the bus 204 can facilitatecommunication between components of the electronic device 200, and in anexample can ease processing performed by the processor 202.

Where the electronic device 200 is a server, it can include a computingdevice that can be capable of sending or receiving signals, e.g., via awired or wireless network, or may be capable of processing or storingsignals, e.g., in memory as physical memory states. The server may be anapplication server that includes a configuration to provide one or moreapplications, e.g., aspects of the systems and methods for interactivevideo generation and rendering, via a network to another device. Also,an application server may, for example, host a Web site that can providea user interface for administration of example aspects of the systemsand methods for interactive video generation and rendering.

Any computing device capable of sending, receiving, and processing dataover a wired and/or a wireless network may act as a server, such as infacilitating aspects of implementations of the systems and methods forinteractive video generation and rendering. Thus, devices acting as aserver may include devices such as dedicated rack-mounted servers,desktop computers, laptop computers, set top boxes, integrated devicescombining one or more of the preceding devices, and the like.

Servers may vary widely in configuration and capabilities, but theygenerally include one or more central processing units, memory, massdata storage, a power supply, wired or wireless network interfaces,input/output interfaces, and an operating system such as Windows Server,Mac OS X, Unix, Linux, FreeBSD, and the like.

A server may include, for example, a device that is configured, orincludes a configuration, to provide data or content via one or morenetworks to another device, such as in facilitating aspects of anexample systems and methods for interactive video generation andrendering. One or more servers may, for example, be used in hosting aWeb site, such as the web site www.microsoft.com. One or more serversmay host a variety of sites, such as, for example, business sites,informational sites, social networking sites, educational sites, wilds,financial sites, government sites, personal sites, and the like.

Servers may also, for example, provide a variety of services, such asWeb services, third-party services, audio services, video services,email services, HTTP or HTTPS services, Instant Messaging (IM) services,Short Message Service (SMS) services, Multimedia Messaging Service (MMS)services, File Transfer Protocol (FTP) services, Voice Over IP (VOIP)services, calendaring services, phone services, and the like, all ofwhich may work in conjunction with example aspects of an example systemsand methods for interactive video generation and rendering. Content mayinclude, for example, text, images, audio, video, and the like.

In example aspects of the systems and methods for interactive videogeneration and rendering, client devices may include, for example, anycomputing device capable of sending and receiving data over a wiredand/or a wireless network. Such client devices may include desktopcomputers as well as portable devices such as cellular telephones, smartphones, display pagers, Radio Frequency (RF) devices, Infrared (IR)devices, Personal Digital Assistants (PDAs), handheld computers,GPS-enabled devices tablet computers, sensor-equipped devices, laptopcomputers, set top boxes, wearable computers, integrated devicescombining one or more of the preceding devices, and the like.

Client devices, as may be used in example systems and methods forinteractive video generation and rendering, may range widely in terms ofcapabilities and features. For example, a cell phone, smart phone ortablet may have a numeric keypad and a few lines of monochromeLiquid-Crystal Display (LCD) display on which only text may bedisplayed. In another example, a Web-enabled client device may have aphysical or virtual keyboard, data storage (such as flash memory or SDcards), accelerometers, gyroscopes, GPS or other location-awarecapability, and a 2D or 3D touch-sensitive color screen on which bothtext and graphics may be displayed.

Client devices, such as client devices 102-106, for example, as may beused in example systems and methods for interactive video generation andrendering, may run a variety of operating systems, including personalcomputer operating systems such as Windows, iOS or Linux, and mobileoperating systems such as iOS, Android, Windows Mobile, and the like.Client devices may be used to run one or more applications that areconfigured to send or receive data from another computing device. Clientapplications may provide and receive textual content, multimediainformation, and the like. Client applications may perform actions suchas browsing webpages, using a web search engine, interacting withvarious apps stored on a smart phone, sending and receiving messages viaemail, SMS, or MMS, playing games (such as fantasy sports leagues),receiving advertising, watching locally stored or streamed video, orparticipating in social networks.

In example aspects of the systems and methods for interactive videogeneration and rendering, one or more networks, such as networks 110 or112, for example, may couple servers and client devices with othercomputing devices, including through wireless network to client devices.A network may be enabled to employ any form of computer readable mediafor communicating information from one electronic device to another. Anetwork may include the Internet in addition to Local Area Networks(LANs), Wide Area Networks (WANs), direct connections, such as through aUniversal Serial Bus (USB) port, other forms of computer-readable media,or any combination thereof. On an interconnected set of LANs, includingthose based on differing architectures and protocols, a router acts as alink between LANs, enabling data to be sent from one to another.

Communication links within LANs may include twisted wire pair or coaxialcable, while communication links between networks may utilize analogtelephone lines, cable lines, optical lines, full or fractionaldedicated digital lines including T1, T2, T3, and T4, IntegratedServices Digital Networks (ISDNs), Digital Subscriber Lines (DSLs),wireless links including satellite links, optic fiber links, or othercommunications links known to those skilled in the art. Furthermore,remote computers and other related electronic devices could be remotelyconnected to either LANs or WANs via a modem and a telephone link.

A wireless network, such as wireless network 110, as in example systemsand methods for interactive video generation and rendering, may coupledevices with a network. A wireless network may employ stand-alone ad-hocnetworks, mesh networks, Wireless LAN (WLAN) networks, cellularnetworks, and the like.

A wireless network may further include an autonomous system ofterminals, gateways, routers, or the like connected by wireless radiolinks, or the like. These connectors may be configured to move freelyand randomly and organize themselves arbitrarily, such that the topologyof wireless network may change rapidly. A wireless network may furtheremploy a plurality of access technologies including 2nd (2G), 3rd (3G),4th (4G) generation, Long Term Evolution (LTE) radio access for cellularsystems, WLAN, Wireless Router (WR) mesh, and the like. Accesstechnologies such as 2G, 2.5G, 3G, 4G, and future access networks mayenable wide area coverage for client devices, such as client deviceswith various degrees of mobility. For example, a wireless network mayenable a radio connection through a radio network access technology suchas Global System for Mobile communication (GSM), Universal MobileTelecommunications System (UMTS), General Packet Radio Services (GPRS),Enhanced Data GSM Environment (EDGE), 3GPP Long Term Evolution (LTE),LTE Advanced, Wideband Code Division Multiple Access (WCDMA), Bluetooth,802.11b/g/n, and the like. A wireless network may include virtually anywireless communication mechanism by which information may travel betweenclient devices and another computing device, network, and the like.

Internet Protocol (IP) may be used for transmitting data communicationpackets over a network of participating digital communication networks,and may include protocols such as TCP/IP, UDP, DECnet, NetBEUI, IPX,Appletalk, and the like. Versions of the Internet Protocol include IPv4and IPv6. The Internet includes local area networks (LANs), Wide AreaNetworks (WANs), wireless networks, and long haul public networks thatmay allow packets to be communicated between the local area networks.The packets may be transmitted between nodes in the network to siteseach of which has a unique local network address. A data communicationpacket may be sent through the Internet from a user site via an accessnode connected to the Internet. The packet may be forwarded through thenetwork nodes to any target site connected to the network provided thatthe site address of the target site is included in a header of thepacket. Each packet communicated over the Internet may be routed via apath determined by gateways and servers that switch the packet accordingto the target address and the availability of a network path to connectto the target site.

The header of the packet may include, for example, the source port (16bits), destination port (16 bits), sequence number (32 bits),acknowledgement number (32 bits), data offset (4 bits), reserved (6bits), checksum (16 bits), urgent pointer (16 bits), options (variablenumber of bits in multiple of 8 bits in length), padding (may becomposed of all zeros and includes a number of bits such that the headerends on a 32 bit boundary). The number of bits for each of the above mayalso be higher or lower.

A “content delivery network” or “content distribution network” (CDN), asmay be used in example systems and methods for interactive videogeneration and rendering, generally refers to a distributed computersystem that comprises a collection of autonomous computers linked by anetwork or networks, together with the software, systems, protocols andtechniques designed to facilitate various services, such as the storage,caching, or transmission of content, streaming media and applications onbehalf of content providers. Such services may make use of ancillarytechnologies including, but not limited to, “cloud computing,”distributed storage, DNS request handling, provisioning, data monitoringand reporting, content targeting, personalization, and businessintelligence. A CDN may also enable an entity to operate and/or manage athird party's Web site infrastructure, in whole or in part, on the thirdparty's behalf.

A Peer-to-Peer (or P2P) computer network relies primarily on thecomputing power and bandwidth of the participants in the network ratherthan concentrating it in a given set of dedicated servers. P2P networksare typically used for connecting nodes via largely ad hoc connections.A pure peer-to-peer network does not have a notion of clients orservers, but only equal peer nodes that simultaneously function as both“clients” and “servers” to the other nodes on the network.

One embodiment of the present invention includes systems, methods, and anon-transitory computer readable storage medium or media tangiblystoring computer program logic capable of being executed by a computerprocessor, related to digital histopathology and microdissection.

As mentioned above, requiring multiple pathologists to review and makedeterminations as to whether a tissue sample (“sample”) is diseased or,in particular, diseased with cancer is unreliable, expensive, and timeconsuming.

An embodiment of the present invention includes determining whether asample is diseased. The embodiment described below refers, inparticular, to cancer. However, embodiments of the present invention maybe used to make a determination as to other diseases.

An embodiment of the present invention relates to determining whether asample is cancerous by using computer vision. Computer vision relates tothe automated extraction, analysis and understanding of usefulinformation from one or more digital images. For example, computervision may be used to determine the age of a person in a photograph bydetermining the location of a face of a person in a digital image,determining the location of the eyes of such person, and measuring theinterpupillary distance of such person.

In the field of machine learning, a Convolutional Neural Network (“CNN”)is an artificial neural network which may be used in the field ofcomputer vision. The article Rethinking the Inception Architecture forComputer Vision by Christian Szegedy et al. (arXiv:1512.00567v3 [cs.CV]11 Dec. 2015) discusses the use of CNNs in computer vision. The CNN hasa plurality of layers, as shown in FIG. 5, and a plurality of parametersin each layer (input size). FIG. 5 includes information on the type oflayer, the patch size and the input size of each layer. The values ofthe parameters determine the output of the CNN.

The CNN may be provided an input of an image of a tissue sample and theCNN may provide, as an output, a probability of whether said image iscancer or non-cancer. The image of the tissue sample may be a slideimage and, in particular, a digital histopathology image. Prior to theCNN making such determination, according to an embodiment of the presentinvention, a CNN may be trained using related images (i.e., images ofcancer cells and images without cancer cells).

FIG. 3 illustrates an architecture diagram of an electronic device thatcan implement one or more aspects of an embodiment of the invention.FIG. 3 includes image processing engine 301. Image processing engine 301may be implemented by programs and/or applications 222 of FIG. 2, whichcan include, for example, software aspects of the digital histopathologyand microdissection system 223. Image processing engine 301 includestraining engine 302, which trains CNN 315.

FIG. 4 illustrates the CNN training process carried out by trainingengine 302. As shown in FIG. 4, training of the CNN 315 by the trainingengine 302 includes a number of steps. In step 401, CNN 315 receives aplurality of patches of digital tissue images of different types/groups,The plurality of patches may, for example, include a plurality of normalpatches and a plurality of positive patches (training patches 302A). Thetraining patches 302A are portions of a larger image. In this case, thelarger image may be a digital image of a biological sample which mayhave positive and normal patches. The training patches may also comefrom multiple larger images. Positive patches are patches which areknown to be cancer and normal patches are patches which are known to benon-cancer (i.e., they may have previously been determined bypathologists or computer vision to be either cancer or non-cancer). Thetypes of cancer may include, but are not necessarily limited to, breastcancer, bladder cancer, brain cancer, lung cancer, pancreatic cancer,skin cancer, colorectal cancer, prostate cancer, stomach cancer, livercancer, cervical cancer, esophageal cancer, leukemia, non-hodgkinlymphoma, kidney cancer, uterine cancer, bile duct cancer, bone cancer,ovarian cancer, gallbladder cancer, gastrointestinal cancer, oralcancer, throat cancer, ocular cancer, pelvic cancer, spinal cancer,testicular cancer, vaginal cancer, vulvar cancer, and thyroid cancer.

In step 401, the training engine 302 may provide as input to the not yettrained classifier of the CNN 315 a large number of normal patches and alarge number of positive patches (training patches 302A) (for example1000, 5000, 10000, 20000, 30000, 40000, 50000, 75000, or 100000 positivepatches and an equal number, an unequal number, or a substantiallysimilar number (such as a number within 1%, 3%, 5% or 10%) of normalpatches) to train the CNN 315 in recognizing patches withcharacteristics similar to the input patches. If there is aninsufficient number of unique normal or positive patches, the trainingengine 302 may duplicate a randomly selected (or patch selected by auser) existing training patch in the particular group of patches (i.e.,positive or normal) and modify the patch. For example, the patch may bemodified by rotating it 90, 180 or 270 degrees and/or the color schemeof the patch may be modified and/or a distortion may be added to thepatch and/or the patch may be converted to greyscale and/or a portion ofthe patch may be cropped out and/or the patch may be flipped and/or thepatch may be resized. Training patches can be subjected to a transformthat can include: rotation, skewing, affine, translation, mirror image,etc. As mentioned above, a random patch may be selected and then arandom modification scheme may be applied. Where a variable is involved(such as degrees rotation), a random number may be used to select thevalue of the variable.

The resulting trained classifier of the CNN 315 may be at least one ofthe following types of classifiers: support vector machine, softmax,decision tree, random forest, k nearest neighbor, Linear and QuadraticDiscriminant Analysis, Ridge Regression. MultiLayer Perceptron (MLP),Hyper-pipes, Bayes net, k-means clustering and/or naïve bayes.

In addition to providing a plurality of normal patches and positivepatches, for each patch, the training engine 302 provides the CNN 315values of the correct output for each patch. For example, a 0 may beprovided if the patch is normal and a 1 is provided if the patch ispositive (i.e., cancer or another disease).

In step 403, the training engine 302 sets, in the CNN 315, an input sizeof one or more fully connected layers of the CNN 315 architecture to anew value, the new value being determined based on a cardinality oftypes of patches in the plurality of patches. For example, in the caseof two types of patches, normal and positive, the cardinality of typesof patches would be 2. More specifically, the input size of the softmaxlayer of the CNN 315, as shown in the last row of FIG. 5, may be set to1×1×2.

In step 405, the training engine 302 populates, in the CNN 315, adistribution of values of parameters of the one or more fully connectedlayers (e.g., CNN parameters 309). The distribution of values may be aGaussian distribution, a Poisson distribution, or a user generateddistribution. The CNN parameters 309 determine how the CNN classifiesbased on its training.

A plurality of patches may then be input by the training engine 302 intothe CNN 315 and the initial class probability scores of each patch aregenerated by the CNN 315 and stored in a memory (first initial classprobability scores of the plurality of patches). The initial classprobability score indicates a probability that a particular patch fallswithin a group of normal patches or a group of positive patches (to makea first classification of each patch). Step 405 sets the firstclassification as the current classification.

In step 407, the training engine 302 adjusts, in the CNN 315, the valuesof the parameters 309 of the one or more fully connected layers.

In step 409, after the adjustment of values of the parameters in step407, a plurality of patches are input by the training engine 302 intothe CNN 315 and class probability scores of each patch are determinedafter adjustment and assigned by CNN 315 and stored in a memory asadjusted class probability scores (to make an adjusted classification ofthe plurality of patches). The class probability score of apre-adjustment (or before the latest adjustment) patch may be referredto as the first initial class probability score and the probabilityscore of a post-adjustment patch may be referred to as the secondinitial class probability score

Then, in step 411, training engine 302 determines whether the adjustedclass probability scores (sometimes referred to as the first initialclass probability scores) of the plurality of patches are more accuratethan the current class probability scores (sometimes referred to as thesecond initial class probability scores) of the plurality of patches.That is, in step 411, it is determined whether the parameters adjustedin step 407 produce more accurate probabilities than did the parametervalues used prior to the adjustment in step 407. The determination ofstep 411 may include determining that a sum of squares of a differencebetween the adjusted class probability scores of the plurality ofpatches and a correct initial class probability scores of the pluralityof patches is lower than a sum of squares of a difference between thecurrent class probability scores of the plurality of patches and thecorrect initial class probability scores of the plurality of patches. Ifthe adjusted class probability scores are determined to be more accuratethan the current class probability scores, then the adjustedclassification is set to be the new current classification. The processcan return to step 407 from step 411 and continue iterating steps407-411. That is, the parameters may be adjusted multiple times to findthe best set of parameters.

Once the CNN has been trained according to the process in FIG. 4 and theoptimal parameters have been set/adjusted, the CNN may then be used todetermine initial class probabilities for patches of images ofbiological samples for which the probabilities are unknown. That is,once the classifier is trained, it is ready for use with “test” patches.Test patches are patches from an actual, live patient's tissue sample.

FIG. 6 shows a method for receiving a digital tissue image of abiological sample and determining the portions thereof likely to havecancer and the likelihood of particular regions within the sample havingcancer. The method is performed using the trained classifier.

In step 601, the image processing engine 301 obtains access to a digitaltissue image of a biological sample. The digital image may in variousforms, for example, SVS, TIFF, VMS, VMU, NDPI, SCN, MRXS, SVSLIDE, BIF,PDF, JPG, BMP, GIF and any other digital image format. Moreover, thedigital image may be located on a server (e.g., one or more servers107-109), it may be a large image (many GB in size), the image may bestored in the cloud and all analysis in FIG. 6 may be performed in thecloud. The cloud may include servers 107-109. However, the steps of FIG.6 may also be performed at one or more client devices 102-106 or acombination of servers 107-109 and/or client devices 102-106. Theprocessing may be parallel and take place on multiple servers.

In step 603, tile generation engine 303 tiles the digital tissue imageinto a collection of image patches 307. Each tile/patch may be, forexample, less than or equal to 1000×1000 pixels, less than or equal to400×400 pixels, less than or equal to 256×256 pixels or any othersuitable number of pixels. The tiling step may be performed iterativelyor in parallel by one or more computers. Tiling may include creatingimage patches that are of a uniform size and a uniform shape. The sizeof the patch may be a function of how the classifier was trained. Forexample, if the classifier/CNN was trained using 400×400 patches, thetile generation engine 303 may tile the image into same size (400×400)patches or, within 1%, 3%, 5%, 10%, 20%, 25%, or 30% of the size ofpatches using which the classifier was trained.

In step 603, the patches 307 may or may not be of a uniform size andshape. For example, one patch may be 400×400 and another patch may be300×300 or 300×200. The patches also need not be squares, they may berectangles, circles, ovals or more complex shapes. Various processes maybe used for tiling such as Penrose tiling, bulk exclusion, and/or boundboxes.

In step 603, the generated patches may be overlapping ornon-overlapping. That is, the same area of the digital image may or maynot be included in more than one tile/patch.

In step 605, the patch identification engine 304 identifies/selects aset of target tissue patches from the tiled patches as a function ofpixel content. For example, identification may include filtering thepatches based on color channels of the pixels within the image patches.For example, the identification may be made as a function of thevariance of the patches. The variance of the patches may be based on thevariance of the Red Green Blue (RGB) channels and/or Hue, Saturation,Value (HSV) and/or Hue Saturation and/or Luminosity (HLS) and/or HueSaturation Intensity (HIS) in a particular patch. This step helps insurethat only patches that include cells are considered. Once step 605 iscomplete, only patches with cells are identified/selected. Such patchesare shown in FIG. 9A (although no cells are shown in the patches of FIG.9A, FIG. 9A is a representative diagram of patches and it is assumedthat each patch in FIG. 9A in fact includes a plurality of stainedcells).

In step 607, prior to sending the request to CNN 315, probabilitydetermination engine 305 may select a particular trained classifier fromthe a priori trained classifiers in CNN 315 according to classifierselection criteria defined according to biological sample metadata boundto the digital tissue image. The biological sample metadata includesdigital information associated with at least one of the following: atissue type, a tissue donor, a scanner, a stain, a staining technique,an identifier of a preparer, an image size, a sample identifier, atracking identifier, a version number, a file type, an image date, asymptom, a diagnosis, an identifying information of treating physician,a medical history of the tissue donor, a demographic information of thetissue donor, a medical history of family of the tissue donor, and aspecies of the tissue donor. Multi-plex immune histo chemistry (IHC) maybe used (for example, technology offered by PerkinElmer; seehttp://www.perkinelmer.com/lab-solutions). The IHC system allows for thegenerating of very complex digital images of tissues. The IHC systemprovides for the capturing of many different wavelengths of light frombiotags that adhere to different types of cells. Once the slide isscanned, the system can synthetically re-create a desired stained slide.Thus, it is possible to use such a system to generate training databased on wavelength of light based on the biotag uses, the type oftarget cells (e.g., tumor cells, normal cells, T-Cells, NK cells,B-cells, etc.). Once trained, it is possible to then use the CNN 315 toidentify regions of interest based on the biotags.

The probability determination engine 305 then transmits each patch inFIG. 9A to CNN 315 (which has been trained, and thus includes a databaseof a priori trained classifiers, as discussed above) with a request toassign an initial class probability score indicating a probability thatthe target tissue patch falls within a class of interest. The class ofinterest may include at least one of the following types of tissue:abnormal tissue, benign tissue, malignant tissue, bone tissue, skintissue, nerve tissue, interstitial tissue, muscle tissue, connectivetissue, scar tissue, lymphoid tissue, fat, epithelial tissue, nervoustissue, and blood vessels. The class of interest may also be eithercancer or non-cancer (i.e., positive or normal). The class of interestmay also be different types of cancers. That is, a probability (between0 and 1) that the input patch is cancer (1 being 100% likelihood thatthe patch contains cancer and 0 being 0% likelihood of the patchcontains cancer). The CNN 315 outputs the probability to probabilitydetermination engine 305. Although FIG. 3 shows direct communicationbetween probability determination engine 305 and CNN 315, there may bemultiple nodes between the two and the CNN may process the request usinga plurality of servers, in series or in parallel.

FIG. 9B is a representative diagram showing the initial classprobability scores of each of 25 representative patches, as determinedby CNN 315 and communicated to probability determination engine 305 byCNN 315. In FIGS. 9A-9F, for ease of reference and description only,column and row numbers are labelled in the drawings so that each patchcan be referred to by identifying the row and column number using thefollowing notation: (column number, row number). As can be seen, forexample, in FIG. 9B, the probability that patch (1, 1) includes cancercells is 0.4, the probability that patch (2, 2) includes cancer is cellsis 0.8, the probability that patch (5, 1) includes cancer is 0.05, theprobability that patch (4, 2) includes cancer is 0.9 and so on. Theseprobabilities are based on the likelihood that a particular patch hascancer cells in isolation and do not take into account the probabilitiesof any other patch in computing the probability of a particular patch.The initial class probabilities of each patch are stored in RAM or othermemory.

In step 609, the classification engine 311 generates a first set oftissue region seed location patches by identifying target tissue patcheshaving initial class probability scores that satisfy a first seed regioncriteria. This first seed region criteria may be considered a locationcriteria. For example, the criteria may be identifying any patches withan initial class probability of 0.9 and above. Using the initial classprobabilities assigned in FIG. 9B, FIG. 9C shows the generated first setof tissue region seed patches. In particular, FIG. 9C shows that thegenerated first set of tissue region seed patches includes patch (2, 4),patch (3, 3), patch (3, 4), and patch (4, 2). The generated first set oftissue region seed patches are representatively indicated in FIG. 9C byunderlining the initial class probability of the patch. Theprobabilities of the first set of tissue region seed patches is storedin RAM or other memory. The seed patches can be considered initial seedlocations around which regions of interest are built.

In step 611, the classification engine 311 generates a second set oftissue region seed patches by identifying target tissue patches havinginitial class probability scores that satisfy a second seed regioncriteria. The processing of step 611 may be performed only near (i.e.,within a predetermined number of neighbors from) the first set of tissueregion patches generated in step 609. This second seed region criteriamay be considered a shape criteria That is, the generated second set oftissue region seed patches will generally form a shape, which is oftencontiguous. For example, the criteria may be identifying any patcheswith an initial class probability of 0.5 and above (the second seedregion criteria is generally lower than and easier to satisfy than thefirst seed region criteria). Using the initial class probabilitiesassigned in FIG. 9B, FIG. 9D shows the generated second set of tissueregion seed patches. In particular, FIG. 9D shows that the generatedsecond set of tissue region seed patches includes patch (1, 3), patch(2, 2), patch (2, 3), patch (2, 4), patch (3, 2), patch (3, 3), patch(3, 4), patch (4, 2), patch (4, 3), patch (5, 2) and patch (5, 3). Thegenerated second set of tissue region seed patches are representativelyindicated in FIG. 9D by showing the initial class probability of thegenerated patch in a larger font size. The second set of tissue regionseed patches is stored in RAM or other memory.

In step 613, the classification engine 311 determines the regions ofinterest and calculates a region of interest score for each patch in thesecond set of tissue region seed patches (generated in step 611) as afunction of initial class probability scores of neighboring patches ofthe second set of tissue region seed patches and a distance to patcheswithin the first set of issue region seed patches. Neighboring patchesmay refer to a first neighbor (adjacent neighbors), second neighbor (onepatch between second neighbor patches), a third neighbor (two patchesbetween third neighbors), or any other level neighbor. A distance may bemeasured either in patches or in pixels. In this step, theclassification engine 311 is refining the scores of each patch in thesecond set of tissue region seed patches based on neighbors.

A Region of Interest (ROI) 313 is a group of one or more connectedpatches. ROIs 313 may be calculated separately for the first set oftissue region seed patches, the second set of tissue region seedpatches, or a combined set of first and second sets of tissue regionseed patches. Two patches are connected if one of its 8 neighbors (4edge neighbors and 4 corner neighbors assuming square or rectangularpatches) are in the same set of tissue region seed patches. Patches mayalso be shapes other than square or rectangular. Patches may be, forexample, polygonal, hexagonal (convex and concave), pentagonal,triangular, octagonal, nonagonal, circular, oval, trapezoidal,elliptical, irregular, and the like, Once one or more ROIs 313 aredetermined, a region of interest score (“ROI score”) for each ROI 313 iscalculated by classification engine 311. The ROI 313 score may be afunction of the size of the ROI 313 (i.e., the number of patches orpixels that comprise the ROI). This scoring method leverages the factthat tumor cells tend to exist in groups. Thus, if a patch has a highprobability of containing a tumor/cancer, and several of its neighborsalso have a high probability of containing a tumor, it is more likelythat this ROI is a tumor and the ROI score reflects this highprobability.

In one embodiment of step 613, the classification engine 311 generates alist of ROIs from the first set of tissue region seed patches bygrouping together connected neighbor patches and computing the centroidfor each ROI 313. This results in a list of ROIs L_high. Theclassification engine 311 also generates a list of ROIs from the set thesecond set of tissue region seed patches by grouping together connectedneighbor patches and computing the centroid for each ROI. This resultsin a list of ROIs L_low. Each of the ROIs in L_high is assigned a scoreas follows. If the size (number of patches) of a patch in L_high is 1,the ROI is assigned a score of 0.2; if the size is 2, the ROI isassigned a score of 0.3; if the size is 3, the ROI is assigned a scoreof 0.4; if the size is 4, the ROI is assigned a score of 0.5; if thesize is 5, the ROI is assigned a score of 0.6; if the size is 6, the ROIis assigned a score of 0.7; if the size is 7, the ROI is assigned ascore of 0.8; if the size is 8, the ROI is assigned a score of 0.9; andif the size is 9 or more, the ROI is assigned a score of 1.0. The abovemapping is an example and a different mapping of size to score may beused (for example, as a function of the size of a patch).

Once the above initial scoring is performed, if an ROI in L_low issufficiently close to an ROI in L_high, the classification engine 311boosts the score of the ROI in L_high. This means that if patches withhigh probability (for example, >=0.9) are surrounded by (or near)patches with a lower but still significant tumor probability (forexample, >=0.5), we have greater confidence that this ROI in L_high is atumor. Sufficiently close may be defined as two ROIs where the distancebetween their centroids is less than a predetermined number of patches,for example, 5, or 6, or 7, or 8, or 9, or 10, or 11, or 12, or 13, or14, or 15.

Score boosting is calculated as follows. If the size of the ROI in L_lowthat is sufficiently close to ROI in L_high is 5 patches, we boost thescore of the ROI in L_high by 0.05, if the size is 10 patches, we boostthe score of the ROI in L_high by 0.10 and if the size is 15 patches, weboost the score of the ROI in L_high by 0.15. Sizes between 5-10 and10-15 are rounded to the nearest size with a defined score boost. Thescore has a ceiling of 1.0 (in case the score is boosted above 1.0). Thefinal output may be the list of ROIs L_high, each with a centroidlocation and a score. The ROI(s) and score(s) may be rendered on adisplay.

The ROI(s) may demarcate different types of masks. The ROI(s) mayinclude object foreground masks, used to separate foreground frombackground in images. The ROI(s) may include, for example, a tissuemask, demarcating areas of tissue and excluding areas without tissue.This may be used to concentrate processing resources to the tissue ROI.The ROI(s) may include a microdissection mask, which may be used inconducting a laser (or other type of) microdissection in order to excisea target ROI for further processing. Only certain ROIs may be used as amicrodissection mask based on the size of the ROI and the quality of theROI. That is, certain ROIs may not be suitable for microdissection (forexample, ROIs that are too small overall or too narrow at certainpoints).

For example, as shown in FIG. 9E, there is a single ROI in L_highincluding patch (2, 4), patch (3, 3), patch (3, 4), and patch (4, 2). Asshown in FIG. 9F, there is also a single ROI in L_low including patch(1, 3), patch (2, 2), patch (2, 3), patch (2, 4), patch (3, 2), patch(3, 3), patch (3, 4), patch (4, 2), patch (4, 3), patch (5, 2), andpatch (5, 3).

The size (number of patches) of the ROI in L_high is 4 so the initialROI score would be 0.5. However, based on the score boosting rulesabove, since the centroids of the ROIs in L_high and L_low are within 10patches, and the size of the ROI in L_low is 11 (patch (1, 3), patch (2,2), patch (2, 3), patch (2, 4), patch (3, 2), patch (3, 3), patch (3,4), patch (4, 2), patch (4, 3), patch (5, 2) and patch (5, 3)) so, afterrounding 11 down to 10, the score is boosted by 0.10 from 0.5 for afinal score of 0.6.

In the alternative, the purpose served by steps 609, 611 and 613 can bemore generally implemented using a conditional random field model, asshown in FIG. 10. The conditional random field model is a conditionalprobability distribution, where dependencies among the input variablesdo not need to be explicitly represented. This is in contrast to theexplicit representation performed in steps 609, 611, and 613. The outputof the conditional random field is a modified probability score thattakes into account the input labels and the relational nature of theinitial class probability scores. Specifically, the relational nature ofthe initial class probabilities is represented by k(f_(i), f_(j)) inFIG. 10, which would, for example, increase when input data x_(j) is faraway, both in terms of location (p) and feature (I), from x_(i).Training of the conditional random field parameters in FIG. 10 isaccomplished by minimizing E(x) over the parameters w and θ, given thelabel data μ and input data for p and I. Inference on new data isaccomplished using an iterative message passing algorithm. The modifiedprobability scores can then be used to generate region of interestshapes and scores. It is noted that in FIG. 10, the symbol u in thebottom line (“u=1 if neighbor is different class . . . ”) is referringto μ in the formula. This method is described in further detail in“Efficient Inference in Fully Connected CRFs with Gaussian EdgePotentials” by Philipp Krahenbuhl et al. (Advances in Neural InformationProcessing Systems 24 (2011) 109-117).

In step 615, the classification engine 311 generates region of interestshapes by grouping neighboring patches based on their region of interestscores.

Once the ROIs are calculated, the classification engine 311 generatesregion of interest shapes by grouping neighboring patches based on theirregion of interest scores.

Once the ROIs are established at the “patch layer” using the steps 609,611 and 613 and/or the Conditional Random Field Model, additionalprocessing may be performed at the “cell layer.” In particular, for eachboundary patch in a shape (i.e., connected patches of the second set oftissue region seed patches), the trained classifier of the CNN 315 isused to classify each cell in a patch as positive or negative using theclassifier of CNN 315 if training information at the cell level isavailable (that is, if there exists an a priori database that wastrained using cells (as opposed to patches)).

In particular, if the classifier of CNN 315 was trained on one cellpatches (small patches that include a single cell or single cell withsmall portions of other cells and non-cells), cells are identified and apatch including a single cell are transmitted to the classifier of CNN315 for classification and a probability of cancer is returned asoutput.

In the alternative, a fully convolutional neural network (FCNN) can beused on each boundary patch to identify the exact boundary line thatdifferentiates tumor and non-tumor cells. In particular, the FCNN willoutput a pixel-wise prediction describing the probability of each pixelcontaining a tumor. During training, a FCNN will learn upsamplingweights to transform activations into pixel-wise predictions. See “FullyConvolutional Networks for Semantic Segmentation” by Jonathan Long etal., including FIG. 1, showing pixel-wise prediction.

As a result of the above “cell layer” processing, some of the boundarypatches of a shape that includes connected patches of the second set oftissue region seed patches will get smaller. For example, with referenceto FIG. 9F, if the left half of patch (1, 3) includes non-cancer cellsand the right half of patch (1, 3) includes cancer cells, following the“cell layer” processing, the shape would shrink and would no longerinclude the left half of patch (1, 3). Thus, the shape of the ROI wouldbe refined by the “cell layer” processing.

FIG. 11 illustrates a diagram showing a region of interest boundarygenerated by an electronic device that implements one or more aspects ofan embodiment of the invention.

There may be other uses for technologies of embodiments of the presentinvention. For example, one such use may be detecting foreground asopposed to background objects. For example, the technology/system may beused in vehicle obstacle avoidance in an autonomous vehicle or partiallyautonomous vehicle. The CNN 315 may be trained using photographs takenby or in the vicinity of a vehicle in the process of being driven. Thetraining would include such images being tiled into patches and eachtraining patch would include data regarding whether the patch is in theforeground or background (e.g., 1.0 if background, 0.0 if foreground).

Once the CNN 315 is trained, it may then be used to determine whetherobjects in patches of images taken by or near a moving vehicle are inthe background or foreground. The system may include a plurality ofcameras mounted on the vehicle or in the vicinity (e.g., on signs,traffic lights, etc.) of the vehicle (and received in real time by thesystem via, for example, wireless telecommunication). The images may beprocessed by the system of the trained CNN 315 to determine whetherpatches of the images are in the background or foreground. That is, thesystem may recognize that a particular object is in the background suchas grass, the sky, buildings, or the road. The system may also determinethat an object is a large distance away from the vehicle. On the otherhand, the system may determine that a particular object is in theforeground such as a nearby vehicle, pedestrian, or pothole. Determiningwhat is in the foreground is useful in that a vehicle would then be ableto determine that it needs to avoid objects in the foreground to avoid acollision but needs avoid objects in the background.

As discussed above, the CNN 315 may be trained on more than twoclasses/types of objects/images. That is, instead of training the CNN315 on only two classes of patches (such as cancer/non-cancer, discussedin detail above), the CNN 315 may be trained using, for example, patchesof cancer grades G1, G2, G3, G4 . . . GN. The CNN 315 would then betrained to identify the probability that a patch is in one of grades G1,G2, G3, G4 . . . GN. This may be accomplished by one of two methods.First, a discrete output method may be used. In the discrete outputmethod, the architecture for the patch level classification is similarto that described above except the final (softmax) layer of the CNN 315,as shown in FIG. 5, would be changed from 2 classes to N classes,allowing the CNN 315 to be trained on N classes. In a case in which theN classes are non-ordered (for example, if the classes were animals suchas dog, cat, pig, etc.), the system would return results for each of theN classes at step 607, and then iterate through steps 609, 611, 613, and615 for each of the N classes.

As an alternative, the continuous output method may be used. In thecontinuous output method, regression may be used in the softmax layerinstead of classification. An example of a regression may be a leastsquare fitting or any curve fitting. For example, if there are 5 classes(cancer grades G1, G2, G3, G4, and G5) we may use a range of 0.0 to 5.0to represent the classes. That is, for example, if the CNN 315determines a patch as likely to be type G1, it may output a floatingpoint number close to 1.0, if the CNN 315 determines a patch as likelyto be type G2, it may output a floating point number close to 2.0, andso on. A value such as 2.1 would indicate that, although the patch islikely the type associated with 2 (G2), it is more likely 3.0 (G3) than1.0 (G1). The continuous classification method is only used with orderedclasses.

The system may also be used in land surveying. For example, the CNN 315may be trained using images/patches of various land and/or waterfeatures (such as buildings, fields, rivers, lakes, etc.). Once the CNN315 is trained, it may then receive and classify a plurality of aerialphotographs and determine whether particular patches of images arelakes, rivers, fields, forests, roads and the like.

The system may also be used to determine whether a particular toothcontains cavities and/or an infection or other issue. The trained CNN315 may receive as input one or more images of a tooth or multiple teethfrom one or more angles and/or X-Rays from one or more angles. Thesystem may then determine, by using the trained CNN 315, whether theseveral patches of such images and/or X-Rays are likely to includecavities.

The system may also be used to analyze X-Rays, MRIs, CTs and the like.For example, the system may be trained on fractured vs. non-fracturedbones and determine whether, for example, an X-Ray image includes afractured bone. The system may be similarly trained on MRI and/or CToutput.

The CNN 315 may also be trained on skin diseases such as melanoma. TheCNN 315 may be trained with positive (melanoma) and non-melanoma(normal) patches and then, once trained, determine whether a section ofa skin biopsy or photograph of the skin may is likely to includemelanoma.

The CNN 315 may also be trained on objects in video games. Each frame ofa rendered video game may have foreground objects and a backgroundscene. The CNN 315 can be trained to differentiate between the two, asdiscussed above. The system may also be used to create masks forAugmented Reality (AR) games. For example, a region around a point ofinterest (e.g., landmark, etc.) may be identified. This region can thenbe masked out and replaced with AR content or other overlay. Moreover,an AI process may be created that learns to play a game based on theregions of interest. The AT process then becomes a non-player entity ina game to challenge a player.

While certain illustrative embodiments are described herein, it shouldbe understood that those embodiments are presented by way of exampleonly, and not limitation. While the embodiments have been particularlyshown and described, it will be understood that various changes in formand details may be made. Although various embodiments have beendescribed as having particular features and/or combinations ofcomponents, other embodiments are possible having a combination of anyfeatures and/or components from any of embodiments as discussed above.

What is claimed is:
 1. A computer implemented method of generating atleast one shape of a region of interest in a digital image, the methodcomprising: obtaining, by an image processing engine, access to adigital tissue image of a biological sample; tiling, by the imageprocessing engine, the digital tissue image into a collection of imagepatches; identifying, by the image processing engine, a set of targettissue patches from the collection of image patches as a function ofpixel content within the collection of image patches; assigning, by theimage processing engine, each target tissue patch of the set of targettissue patches an initial class probability score indicating aprobability that the target tissue patch falls within a class ofinterest, the initial class probability score generated by a trainedclassifier executed on each target tissue patch; generating, by theimage processing engine, a first set of tissue region seed patches byidentifying target tissue patches having initial class probabilityscores that satisfy a first seed region criteria, the first set oftissue region seed patches comprising a subset of the set of targettissue patches; generating, by the image processing engine, a second setof tissue region seed patches by identifying target tissue patcheshaving initial class probability scores that satisfy a second seedregion criteria, the second set of tissue region seed patches comprisinga subset of the set of target tissue patches; calculating, by the imageprocessing engine, a region of interest score for each patch in thesecond set of tissue region seed patches as a function of initial classprobability scores of neighboring patches of the second set of tissueregion seed patches and a distance to patches within the first set ofissue region seed patches; and generating, by the image processingengine, one or more region of interest shapes by grouping neighboringpatches based on their region of interest scores.
 2. The method of claim1, wherein the step of tiling the digital tissue image includes creatingimage patches that are of a uniform size and a uniform shape.
 3. Themethod of claim 2, wherein the image patches of uniform size and uniformshape include square patches of less than or equal to 1,000 pixels by1,000 pixels.
 4. The method of claim 3, wherein the square patches areless than or equal to 400 pixels by 400 pixels.
 5. The method of claim4, wherein the square patches are less than or equal to 256 pixels by256 pixels.
 6. The method of claim 1, wherein the step of tiling thedigital tissue image includes creating image patches that are ofnon-uniform size and shape.
 7. The method of claim 1, wherein thecollection of image patches includes non-overlapping patches.
 8. Themethod of claim 1, wherein the step of identifying the set of targettissue patches includes filtering the collection of image patches basedon color channels of pixels within the image patches.
 9. The method ofclaim 8, further comprising filtering image patches of the collection ofimage patches as a function of variance with respect to the colorchannels.
 10. The method of claim 1, wherein the trained classifierincludes a trained neural network.
 11. The method of claim 1, whereinthe trained classifier includes a trained implementation of at least oneof the following types of classifiers: support vector machine, softmax,decision tree, random forest, k nearest neighbor, Linear and QuadraticDiscriminant Analysis, Ridge Regression. Multilayer Perceptron (MLP),Hyper-pipes, Bayes net, k-means clustering and naive bayes.
 12. Themethod of claim 1, further comprising causing a computing device torender the region of interest shapes on a display.
 13. The method ofclaim 1, wherein the region of interest shapes includes at least onetissue mask.
 14. The method of claim 13, wherein the at least one tissuemask comprises a microdissection mask.
 15. The method of claim 1,wherein the class of interest comprises at least one cancer class. 16.The method of claim 15, wherein the at least one cancer class includesone of the following types of cancer: breast cancer, bladder cancer,brain cancer, lung cancer, pancreatic cancer, skin cancer, colorectalcancer, prostate cancer, stomach cancer, liver cancer, cervical cancer,esophageal cancer, leukemia, non-hodgkin lymphoma, kidney cancer,uterine cancer, bile duct cancer, bone cancer, ovarian cancer,gallbladder cancer, gastrointestinal cancer, oral cancer, throat cancer,ocular cancer, pelvic cancer, spinal cancer, testicular cancer, vaginalcancer, vulvar cancer, and thyroid cancer.
 17. The method of claim 1,wherein the class of interest comprises at least one of the followingtypes of tissue: abnormal tissue, benign tissue, malignant tissue, bonetissue, skin tissue, nerve tissue, interstitial tissue, muscle tissue,connective tissue, scar tissue, lymphoid tissue, fat, epithelial tissue,nervous tissue, and blood vessels.
 18. The method of claim 1, whereinthe digital tissue image comprises a slide image of a tissue sampleslide.
 19. The method of claim 18, wherein the slide image comprises adigital histopathology image.
 20. The method of claim 1, furthercomprising obtaining access to a database of a priori trainedclassifiers.
 21. The method of claim 20, further comprising selectingthe trained classifier from the a priori trained classifiers accordingto classifier selection criteria defined according to biological samplemetadata bound to the digital tissue image.
 22. The method of claim 21,wherein the biological sample metadata includes digital informationassociated with at least one of the following: a tissue type, a tissuedonor, a scanner, a stain, a staining technique, an identifier of apreparer, an image size, a sample identifier, a tracking identifier, aversion number, a file type, an image date, a symptom, a diagnosis, anidentifying information of treating physician, a medical history of thetissue donor, a demographic information of the tissue donor, a medicalhistory of family of the tissue donor, and a species of the tissuedonor.
 23. The method of claim 1, wherein calculating a region ofinterest score for each patch in the second set of tissue region seedpatches as a function of neighboring patches includes calculating aconditional random field (CRF) among the neighboring patches.
 24. Themethod of claim 1, wherein calculating a region of interest score foreach patch in the second set of tissue region seed patch as a functionof neighboring patches includes evaluating nearest neighbors.
 25. Themethod of claim 1, wherein the region of interest shapes includes shapescomposed of patches.
 26. The method of claim 1, wherein the region ofinterest shapes comprise shape sub-patch level features.
 27. The methodof claim 26, wherein the image data comprises tissue image data and theregion of interest shapes comprise sub-patch level shapes at the celllevel.
 28. The method of claim 26, wherein the step of generating theregion of interest shapes includes classifying cells by cell-type withinpatches in the second set of tissue region seed patches according to acell-level trained neural network.
 29. The method of claim 27, whereinthe step of generating the region of interest shapes includesdelineating a boundary between cells via a trained fully convolutionalneural network.
 30. The method of claim 29, wherein the trained fullyconvolutional neural network is trained at a patch level on known celltypes within training patches.
 31. The method of claim 1, wherein theassigning the initial class probability score of a target tissue patchis performed without consideration of the initial class probability ofany other patch of the set of target tissue patches.
 32. The method ofclaim 1, wherein neighboring patches of the second set of tissue regionseed patches comprises one or more patches adjacent to at least onepatch of the second set of tissue region seed patches.
 33. The method ofclaim 1, further comprising, training the trained classifier by a methodcomprising: receiving, by a convolutional neural network, a plurality ofpatches of digital tissue images, the plurality of patches including aplurality of normal patches and a plurality of positive patches;setting, by the convolutional neural network, an input size of one ormore fully connected layers of an architecture of the convolutionalneural network to a new value, the new value being determined based on acardinality of types of patches in the plurality of patches; populating,by the convolutional neural network, a distribution of values ofparameters of the one or more fully connected layers; adjusting, by theconvolutional neural network, the values of the parameters of the one ormore fully connected layers; assigning, by the convolutional neuralnetwork, an initial class probability score to each of the plurality ofpatches, the initial class probability score indicating a probabilitythat the target tissue patch falls within a group of normal patches or agroup of positive patches; and determining whether first initial classprobability scores of the plurality of patches are more accurate thansecond initial class probability scores of the plurality of patches. 34.The method of claim 33, wherein the plurality of digital tissue imagesincludes a cardinality of normal patches that is substantially similarto a cardinality of positive patches.
 35. The method of claim 33,wherein the determining that the first initial class probability scoresof the plurality of patches are more accurate than the second initialclass probability scores of the plurality of patches comprisesdetermining that a sum of squares of a difference between the firstinitial class probability scores of the plurality of patches and correctinitial class probability scores of the plurality of patches is lowerthan a sum of squares of a difference between the second initial classprobability scores of the plurality of patches and the correct initialclass probability scores of the plurality of patches.
 36. The method ofclaim 35, wherein the assignment of the second initial class probabilityscores of the plurality of patches occurs earlier than the assignment ofthe first initial class probability scores of the plurality of patches.37. The method of claim 33, wherein the one or more fully connectedlayers comprise a softmax layer.
 38. The method of claim 33, wherein thetypes of patches are the normal patches and the positive patches. 39.The method of claim 33, wherein the new value comprises 1×1×2.
 40. Themethod of claim 33, wherein the distribution of values of parameters ofthe one or more fully connected layers is a Gaussian distribution.
 41. Acomputer implemented method of training a classifier for generating atleast one shape of a region of interest in a digital image, the methodcomprising: receiving, by a convolutional neural network, a plurality ofpatches of digital tissue images, the plurality of patches including aplurality of normal patches and a plurality of positive patches;setting, by the convolutional neural network, an input size of one ormore fully connected layers of an architecture of the convolutionalneural network to a new value, the new value being determined based on acardinality of types of patches in the plurality of patches; populating,by the convolutional neural network, a distribution of values ofparameters of the one or more fully connected layers; adjusting, by theconvolutional neural network, the values of the parameters of the one ormore fully connected layers; assigning, by the convolutional neuralnetwork, an initial class probability score to each of the plurality ofpatches, the initial class probability score indicating a probabilitythat the target tissue patch falls within a group of normal patches or agroup of positive patches; and determining whether first initial classprobability scores of the plurality of patches are more accurate thansecond initial class probability scores of the plurality of patches. 42.A computer implemented method of generating at least one shape of aregion of interest in a digital image, the method comprising: obtaining,by an image processing engine, access to a digital tissue image of abiological sample; tiling, by the image processing engine, the digitaltissue image into a collection of image patches, the collection of imagepatches comprising a set of target tissue patches; assigning, by theimage processing engine, each target tissue patch of the set of targettissue patches an initial class probability score indicating aprobability that the target tissue patch falls within a class ofinterest, the initial class probability score generated by a trainedclassifier executed on each target tissue patch; generating, by theimage processing engine, a first set of tissue region seed patches byidentifying target tissue patches having initial class probabilityscores that satisfy a first seed region criteria, the first set oftissue region seed patches comprising a subset of the set of targettissue patches; generating, by the image processing engine, a second setof tissue region seed patches by identifying target tissue patcheshaving initial class probability scores that satisfy a second seedregion criteria, the second set of tissue region seed patches comprisinga subset of the set of target tissue patches; calculating, by the imageprocessing engine, a region of interest score for each patch in thesecond set of tissue region seed patches as a function of initial classprobability scores of neighboring patches of the second set of tissueregion seed patches and a distance to patches within the first set ofissue region seed patches; and generating, by the image processingengine, one or more region of interest shapes by grouping neighboringpatches based on their region of interest scores.
 43. A computerimplemented method of generating at least one shape of a region ofinterest in a digital image, the method comprising: obtaining, by animage processing engine, access to a digital tissue image of abiological sample; tiling, by the image processing engine, the digitaltissue image into a collection of image patches; obtaining, by the imageprocessing engine, a plurality of features from each patch in thecollection of image patches, the plurality of features defining a patchfeature vector in a multidimensional feature space including theplurality of features as dimensions; assigning, by the image processingengine, each patch in the collection of image patches an initial classprobability score indicating a probability that the patch falls within aclass of interest, the initial class probability score generated by atrained classifier executed on each patch in the collection of imagepatches using the plurality of features of each patch in the collectionof image patches; generating, by the image processing engine, a firstset of tissue region seed patches by identifying patches in thecollection of image patches having initial class probability scores thatsatisfy a first criteria; generating, by the image processing engine, asecond set of tissue region seed patches by identifying patches in thecollection of image patches having initial class probability scores thatsatisfy a second criteria; and generating, by the image processingengine, one or more region of interest shapes as a function of, for eachpatch in the collection of image patches, its inclusion in the first setof tissue region seed patches and the second set of tissue region seedpatches.
 44. A digital image region of interest generation system, thesystem comprising: a database configured to store a digital tissue imageof a biological sample; an image processing engine configured to: obtainaccess to the digital tissue image of the biological sample; tile thedigital tissue image into a collection of image patches; identify a setof target tissue patches from the collection of image patches as afunction of pixel content within the collection of image patches; assigneach target tissue patch of the set of target tissue patches an initialclass probability score indicating a probability that the target tissuepatch falls within a class of interest, the initial class probabilityscore generated by a trained classifier executed on each target tissuepatch; generate a first set of tissue region seed patches by identifyingtarget tissue patches having initial class probability scores thatsatisfy a first seed region criteria, the first set of tissue regionseed patches comprising a subset of the set of target tissue patches;generate a second set of tissue region seed patches by identifyingtarget tissue patches having initial class probability scores thatsatisfy a second seed region criteria, the second set of tissue regionseed patches comprising a subset of the set of target tissue patches;calculate a region of interest score for each patch in the second set oftissue region seed patches as a function of initial class probabilityscores of neighboring patches of the second set of tissue region seedpatches and a distance to patches within the first set of issue regionseed patches; and generate one or more region of interest shapes bygrouping neighboring patches based on their region of interest scores.45. A digital image region of interest generation system, the systemcomprising: a database configured to store a digital tissue image of abiological sample; an image processing engine configured to: obtainaccess to the digital tissue image of the biological sample from thedatabase; tile the digital tissue image into a collection of imagepatches; obtain a plurality of features from each patch in thecollection of image patches, the plurality of features defining a patchfeature vector in a multidimensional feature space including theplurality of features as dimensions; assign each patch in the collectionof image patches an initial class probability score indicating aprobability that the patch falls within a class of interest, the initialclass probability score generated by a trained classifier executed oneach patch in the collection of image patches using the plurality offeatures of each patch in the collection of image patches; generate afirst set of tissue region seed patches by identifying patches in thecollection of image patches having initial class probability scores thatsatisfy a first criteria; generate a second set of tissue region seedpatches by identifying patches in the collection of image patches havinginitial class probability scores that satisfy a second criteria; andgenerate one or more region of interest shapes as a function of, foreach patch in the collection of image patches, its inclusion in thefirst set of tissue region seed patches and the second set of tissueregion seed patches.